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PREFACE 

This  paper  describes  the  Subjective  Transfer  Function  (STF)  approach 
to  analyzing  systems  that  was  developed  during  research  on  evaluating  the 
contribution  of  command  and  control  to  the  overall  combat  effectiveness  of 
tactical  air  forces.  The  purpose  of  the  paper  is  to  explain  the  advantages 
of  using  the  STF  approach  over  other  commonly  used  approaches  in  terms  of 
developing  causal  system  models,  and  to  detail  the  steps  involved  in  its 
use  in  situations  where  hard  (equipment) ,  soft  (procedures)  and  human 
elements  all  have  main  effects  on  outcomes. 


THE  SUBJECTIVE  TRANSFER  FUNCTION  APPROACH  FOR  ANALYZING  SYSTEMS 


The  subjective  transfer  function  (STF)  approach  was  developed  for 
analyzing  complex  systems  where  many  variables  either  directly  or 
indirectly  impact  on  system  outcomes.  The  idea  is  to  test  hypotheses 
about  how  "experts"  perceive  their  system  to  function.  Hypotheses  are 
algebraic  functions  (STFs)  that  specify  how  factors  comprising  the 
system  affect  judged  outcomes.  Once  the  STFs  are  known  for  all  groups 
of  system  experts,  they  can  be  used  to  evaluate  how  changes  in  system 
inputs  affect  outcomes.  We  detail  the  steps  involved  in  the  STF 
approach,  discuss  its  measurement  basis,  and  provide  illustrations 
from  research  in  tactical  Air  Force  command  and  control. 


Based  on  research  for  Project  AIR  FORCE  (See  Refs.  37-39),  with 
additional  support  from  The  Rand  Corporation.  The  authors  thank 
Barbara  Rose  for  her  research  contributions. 
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i.  INTRODUCTION 


The  Subjective  Transfer  Function  (STF)  approach  is  a  subjective 
measurement  method  for  analyzing  complex  systems  whore  many  factors 
either  directly  or  indirectly  impact  on  system  outcomes  (37).  In  this 
approach,  systems  are  analyzed  from  the  perspective  of  the  "expert,"  who 
by  definition  knows  and  understands  the  system.  The  expert  judges  what 
outcomes  would  result  from  different  descriptions  of  system 
capabilities.  The  measurement  problem  is  one  of  constructing  these 
descriptions  from  experimental  designs  that  allow  effects  of  the 
information  contained  in  the  descriptions  on  judged  outcomes  to  be 
measured.  The  algebraic  modeling  approach  to  subjective  measurement 
provides  this  capability.  This  approach  is  the  measurement  basis  of  the 
STF  approach. 

In  this  paper,  we  first  discuss  ^the  features  of  the  algebraic 
modeling  approach  and  describe, how  these  features  provide  resolutions  to 
measurement  problems  found  with  other  commonly  used  approaches.  Then  we' 
describe  how  the  STF  approach  incorporates  basic  features  of  the 
algebraic  modeling  approach  and  provides  additional  features  to  handle 
complex  systems.  We  use  examples  from  research  in  tactical  air  command 
and  control  to  illustrate  the  steps  involved  in  applying  the  STF 
approach . 
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II.  THE  ALGEBRAIC  MODELING  APPROACH  AND  ITS  ADVANTAGES 


The  algebraic  modeling  approach  to  subjective  measurement  includes 

functional  measurement  (2,  3,  4)  conjoint  measurement  (20,  21),  and  the 

principle  of  stimulus  scale  convergence  and  scale-free  tests  (5,  13, 

14),  which  are  important  tools  of  model  analysis.  A  discussion  of  the 

basic  ideas  of  the  algebraic  modeling  approach  can  be  facilitated  by 

breaking  up  judgments  into  three  parts.  First,  stimuli  are  transformed 

to  their  subjective  counterparts.  For  two  stimuli,  S(i)  and  S(j),  this 

conversion  process  can  be  written 

s  =  H(S  )  (1) 

i  i 

and 

s  =  H(S  ), 
j  j 

where  H  is  the  transformation  function  (referred  to  the  psychophysical 
function  or  utility  curve),  and  s(i)  and  s(j)  are  the  subjective  scale 
values.  Second,  subjective  scales  values  are  combined  into  a  subjective 
response,  r,  by  the  combination  function,  C: 

r  =  C(s  ,s  ).  (2) 

j  j 

Third,  the  subjective  response  is  transformed  into  an  overt  response,  R, 


by  the  judgment  function,  J. 


R  =  J(r). 


(3) 


In  the  algebraic  modeling  approach,  the  idea  is  to  design 
experiments  that  provide  str  ingtent  tests  of  the  hypothesized 
combination  function,  C.  Once  the  unique  predictions  of  an  hypothesized 
function  are  supported  by  the  data,  the  subjective  values  (s  and  r  in 
the  above  equations)  are  known;  they  arc  the  least-squares  estimates  of 
the  function.  Thus  in  this  approach,  the  validity  of  the  subjective 
measures  rests  on  the  validity  of  the  combination  function,  C.  If  an 
hypothesized  function  model  does  not  pass  stringent  tests  of  its  unique 
predictions,  then  both  the  function  and  its  scale  values  are  rejected. 

Carefully  constructed  experimental  designs  are  required  to 
adequately  test  among  the  unique  predictions  of  the  functions  under 
consideration.  A  detailed  discussion  of  these  designs  is  beyond  the 
scope  of  this  paper.  Examples  of  factorial  combinations  of  stimuli  that 
allow  tests  betwen  the  predictions  of  additive  and  interactive  functions 
can  be  found  in  Anderson  (4),  Birnbaum  and  Veit  (13),  and  Levin  (16). 
Descriptions  of  different  predictions  of  polynomial  functions  can  be 
found  in  Krantz  et  al.  (20);  examples  of  the  scale-free  design  that 
allows  a  test  of  the  linearity  of  J  in  Eq .  2  can  be  found  in  Birnbaum 

(5) ,  Birnbaum  and  Veit  (14),  and  Veit,  Rose,  and  Ware  (40).  Birnbaum 

(6)  and  Norman  (24)  discuss  some  problems  and  resolutions  involved  in 
testing  weighted  functions  that  contain  a  weight  and  scale  value  product 
(for  example,  the  multiple  regression  model  or  an  averaging  model  such 
as  the  multi-attribute  utility  function).  Examples  of  experimental 


designs  that  allow  tests  of  the  predictions  of  these  functions  can  be 
found  in  Birnbaum  (6),  Birnbaum  and  Stegner  (11,  12),  and  Veit,  Cal  loro. 
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and  Rose  ( 38 ) . 

A  major  problem  with  some  subjective  measurement  approaches  is  that 
they  do  not  employ  proper  experimental  des igns - -those  that  allow  tests 
of  the  assumed  function's  predictions.  Thus,  conclusions  about  effects 
of  factors  and  subjective  scale  values  are  highly  suspect.  Two  commonly 
used  approaches  that  fall  into  this  category  are  discussed  next. 

US  I  No  'DIRECT'  SCALES  AND  AS  St. 'MED  MODELS 

Many  researchers  using  the.  multiple  regression  or  multi-attribute 
utility  theory  function  assume  the  validity  of  their  function  as  the 
appropriate  combination  function  (C),  as  well  as  the  validity  of  the 
weights  and  scale  values  they  use  to  compute  the  function.  Neither 
assumption  can  be  tested  using  their  experimental  designs. 

Subjective  measures  of  weights  and  scale  values  are  generally 
obtained  using  "direct"  scaling  techniques  (31,  32).  In  the  direct 
approach,  scales  of  "sensation"  are  obtained  by  having  respondents 
assign  numbers  to  stimulus  descriptions  according  to  instructions.  Many 
oporational  definitions  have  been  proposed  for  obtaining  direct  scales 
(for  example,  the.  category  rating  and  magnitude  estimation  scales 
recommended  by  Gardiner  and  Kdwards  (17)  for  obtaining  scales  for  the 
subjective  expected  utility  mode  1  )  .  However,  different  operational 
definitions  of  the  same  stimuli  produce  different  "scales."  Further, 
scale  values  depend  on  features  of  the  experimental  design,  for  example, 
the  range  and  spacing  of  the  stimuli  and  the  response  scale.  More 
importantly,  psychologists  have  severely  questioned  the  scientific 
meaningfulness  of  "scale*  values"  obtained  from  operational  definitions 


t 
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(9,  13,  19,  29,  30,  34,  35).  Despite  tin*  lack  of  a  testable  basis  for 
conclusions,  many  people,  interested  in  obtaining  subjective  measures  for 
a  wide  variety  of  reasons  (e.g.,  analyzing  systems)  use  and  recommend 
these  procedures  (see  as  examples,  17,  18,  25,  22,  2b,  and  the  PATTERN 
technique  described  in  Waddington  (41)). 

SAATY ' S  APPROACH 

Saaty  (28)  proposes  another  approach  to  obtain  subjective  measures 
that  is  also  fraught  with  measurement  problems:  his  ratio  model  is  not 
adequately  tested  and  therefore  his  scale  values  are  suspect,  and  he 
doesn't  provide  any  assessment  of  his  aggregation  model. 

Saaty  proposes  that  a  ratio  model  underlies  category  ratings  of 
ratios.  This  is  a  testable  proposition  using  the  factorial  design  of 
stimulus  cues  that  he  describes  (given  that  the  respondent  and  not  the 
experimenter  fills  in  the  entire  matrix).  However,  the  appropriate  test 
of  the  model  is  not  the  goodness-of- f it  index  he  recommends.  Indices  of 
fit  can  be  high  when  deviations  are  significant  and  systematic.  A  way 
for  both  the  researcher  and  the  reader  to  assess  the  fit  of  a  ratio 
model  is  to  see  graphs  of  the  data.  When  responses  are  plotted  as  a 
function  of  the  levels  of  one  factor  with  a  separate  curve  for  each 
level  of  the  other  factor,  the  resulting  curves  should  form  a  bilinear 
fan;  deviations  from  the  bilinear  prediction  of  the  model  (which  can  bo 
obtained  from  the  analysis  of  variance)  should  be  nons igni f icant  (2,  13, 
35).  We  have  graphed  the  data  presented  in  Saaty  (28)  and  have  found 
them  to  be  considerably  different  in  form  than  the  ratio  model's 
predictions.  If  the  model  does  not  account  for  the  data,  what  meaning 
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can  be  attributed  to  the  scale  values  derived  from  the  model? 

Further,  suppose  a  ratio  model  did  account  for  overt  responses, 
that  is,  deviations  from  bilinearity  were,  nonsignificant.  There  is  a 
body  of  literature  (7,  8,  10,  35,  36)  that  suggests  that  respondents 
take  differences  when  instructed  to  take  ratios  and  then  take  an 
exponential  transformation  of  the  differences.  That  is,  when 
respondents  are  instructed  to  make  ratio  judgments,  the  combination  rule 
(C  in  Eq .  2  above)  is  a  subtractive  model  and  J  in  Eq .  3  is  an 
exponential  transformation.  Thus,  the  stimulus  scale  values  would  have 
to  be  obtained  from  the  subtractive  model  after  performing  a  logarithmic 
transformation  on  the  "ratios".  The  scale  values  derived  from  the 
subtractive  model  would  be  unique  to  an  interval  scale.,  since  any  linear 
transformation  of  the  scale  values  used  in  the  subtractive  model  would 
reproduce  the  rank  order  of  the  data  points  in  a  factorial  matrix.  It 
is  interesting  to  note  that  scale  values  derived  from  a  ratio  model  are 
unique  only  to  a  power  transformation  (20).  Thus,  even  if  the  more 
stringent  designs  were  employed  (see  for  example,  7,  8,  and  35).  that 
provided  a  test  of  whether  C  in  Eq .  2  was  subtractive  or  ratio  and  the 
data  supported  a  ratio  model,  the  scale  values  derived  from  the  ratio 
model  would  only  be  a  power  trails  format  ion  of  the  "true."  values  under 
the  model;  they  would  not  be  ratio  scales  as  Saaty  suggests.  Therefore, 
it  would  be  inappropriate  to  make  ratio  comparisons  of  the  "weights" 


(see,  for  example,  ( 1 ) ) . 
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SUMMARY  REMARKS 

The  algebraic  modeling  approach  places  subjective  measurement  in  a 
scientific  framework  in  the.  sense  of  hypothesis  testing  and  rejecting. 
Stimulus  scales  are  parameters  of  a  theoretical  function  that  accounts 
for  a  nontrivial  data  array. 
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III.  THE  STF  Al’PKOACH 


Complex  systems  arc  defined  here*  ns  those  having  many  sots  of 
factors  that  impact  on  system  outcomes  either  (a)  directly  or  (b) 
indirectly  by  affecting  outcomes  internal  to  the  system.  Different 
groups  of  people  are  generally  "expert"  about  different  parts  of  the 
system.  In  the  subjective  transfer  function  (STF)  approach,  the  effects 
of  system  factors  on  judged  outcomes  are  measured  for  each  group  of 
experts  separately.  The  STFs  serve  to  bring  the  perceptions  of  all  of 
the  groups  together  into  a  cohesive  whole  that  makes  it  possible  to 
compare  effects  of  different  system  capabilities  on  the  outcomes  of 
interest.  These  ideas  are  illustrated  in  this  section. 

Three  major  stages  characterize  the  STF  approach:  (a)  developing 
initial  system  structures,  (b)  testing  among  hypothesized  STFs  to 
explain  the  causal  relationship  between  factors  defining  the  system  and 
the  outcomes  they  affect,  and  (c)  evaluating  different  systems  defined 
by  the  same  structure  but  that  vary  in  their  system  capabilities.  These 
three  stages  are  discussed  below. 

DEV ELOPING  A  STRUCTURE  OF  THE  SYSTEM 

A  structure  of  the  system  identifies  the  factors  and  outcomes  of 
interest.  Structure  development  is  done  in  conjunction  with  the 
appropriate  body  of  system  experts. 


I dent i fy ing  Outcomes,  Factors,  and  Factor _ Levels 

The  first  step  in  developing  a  structure  is  to  identify  the 
outcomes  produced  by  tin-  system  that  provide  the  important  external 
measures  of  the  system's  effectiveness.  This  is  done  through 
interaction  with  those  interested  in  the  system  evaluation.  Then 
factors  thought  to  directly  affect  these  outcomes  are  identified.  These 
factors  may  be  outcomes  that  are  produced  within  the  system  (referred  to 
as  suboutcomes)  that  are  themselves  affected  by  other  system  factors. 
These  f actor/outcome  (suboutcome)  sets,  which  are  called  experimental 
units,  are  identified  through  interaction  with  people  that  are  expert  in 
the  areas.  A  hierarchical  causal  representation  of  the  system  develops 
by  specifying  system  factors  for  suboutcomes  until  all  suboutcome.s  are 
affected  only  by  factors  that  represent  system  input  characteristics  or 
basic  system  features.  Such  factors  are  called  "primitive  factors." 

A  complex  system  that  includes  two  experimental  units  is  depicted 
in  Fig.  1.  This  is  a  structure  of  a  tactical  Air  Force  command  and 
control  system  that  was  investigated  using  the  STF  approach  (38).  In 
this  figure,  there  is  one  external  outcome  measure- -how  well  Air  Force 
commanders  can  perform  their  immediate  target  ing  task  (pairing  tactical 
aircraft  with  important  enemy  ground  force  targets  in  a  timely  manner); 
there  is  also  one  factor/outcome  set  (experimental  unit  l--upper  portion 
of  the  figure),  and  one  factor/suboutcome  set  (experimental  unit  2-- 
lower  portion  of  the  figure).  The  expert  group  corresponding  to 
experimental  unit  1  performs  the  immediate  targeting  task.  The  group 
corresponding  to  experimental  unit  two  is  expert  in  identifying  enemy- 


targets  . 


In  the  first  experimental  unit,  six  factors  are  hypothesized 
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to  directly  impact  on  Immediate  Targeting.  In  the  second  experimental 
unit,  seven  factors  are  hypothesized  to  impact  on  Immediate  Targeting 
indirectly,  through  the  suboutcome  Target  Identification.  Thus,  the 
hypothesis  shown  in  Fig.  1  says  that  part  of  the  ability  to  do  immediate 
t urge 1  ing  depends  on  how  well  the  targeteers  are  able  to  identify 
important  enemy  targets. 

Simultaneous  with  identifying  the  factors,  dimensions  associated 
with  factors  and  outcomes  have  to  be  defined.  The  definitions  for  the 
outcomes  and  factors  shown  in  Fig.  1  are  presented  for  each  experimental 
unit  separately  in  Table  1;  definitions  for  experimental  units  one  and 
two  are  shown  in  Panels  A  and  B,  respectively.  (Note  that  outcomes 
define  the  judgment  task;  factors  define  system  capabilities.)  Factor 
levels  selected  for  experimental  manipulation  should  span  the  range  from 
the  worst  to  the  best  capability  that  might  be  expected  over  a  given 
time  period.  This  feature  is  important  if  future  conditions  or  system 
characteristics  are  to  be  built  into  the  model  for  evaluation  purposes. 
Factor  levels  that  were  selected  for  the  factors  shown  in  Fig.  1  are 
presented  next  to  each  factor  definition  in  Table  1. 

The  structural  representation  shown  in  Fig.  1  hypothesizes  two 
STFs.  The  first  (Tl)  specifies  the  causal  link  among  the  six  factors 
impacting  on  the  immediate  targeting  outcome,  and  the  second  (T2) 
specifies  the  causal  link  among  the  seven  factors  impacting  on  Target 
Identification.  These  functions  are  referred  to  as  subject ive  because 
they  are  models  of  combination  processes  (C  in  Eq .  2)  not  directly 
observed.  They  are  referred  to  as  transfer  functions  because,  when 


their  functional  forms  have  been  determined  and  they  are  being  computed 
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to  evaluate  a  particular  system,  the  output  of  one  function  is 
transferred  for  use  (after  conversion  to  a  subjective  value)  as  an  input 
value  to  the  function  above  it.  for  example,  the  output  of  T2  in  Fig.  1 
would  identify  the  factor  level  and  lienee  the  subjective  input  value  for 
Target  Identification  needed  to  compute  T1 .  This  is  why  it  is  important 
to  identify  factors  that  link  experimental  units  in  the  same  terms  (note 
in  Table  1  that  immediate  targeting  is  defined  in  the  same  terms  when  it 
is  a  dependent  variable  (experimental  unit  2)  and  an  independent 
variable  (experimental  unit  1).  Examples  of  using  STFs  are  presented 
later . 

Identifying  Alternative  Structures 

Structural  alternatives  refer  to  alternative  hypotheses  about  the 
number  of  STF  paths  linking  factors  to  outcomes.  Alternative  structural 
hypotheses  arise  for  two  reasons.  The  first  reason  concerns  different 
hypotheses  about  how  the  expert  combines  information  included  in  a 
description  of  a  system's  capabilities.  For  example,  it  seems 
reasonable  to  hypothesize  that  an  immediate  targeting  expert 
(experimental  unit  1  of  Fig.  1)  might  combine  information  about  the 
three  factors  concerning  their  friendly  forces  (Alert  Forces,  Airborne 
Forces,  and  Weather)  separately  from  information  about  their  other 
capabilities--Target  Identification,  Facility  Operability,  and 
Dissemination.  The  upper  portion  of  Fig.  2  shows  that  this  separate 
combination  process  is  represented  structurally  by  inserting  an 
"intermediary  factor"--Execution  Status  In  format  ion- - into  the  structure. 
(An  intermediary  factor  is  not  identified  by  factor  levels  because  it  is 
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not  manipulated  in  experimental  designs.)  This  represents  the  hypothesis 
that  respondents  combine  information  about  their  own  forces  and  then 
combine  this  subjective  output  with  their  values  associated  with  Target 
Ident i f icat ion ,  Facility  Operability,  and  Dissemination  before  making 
their  immediate  targeting  judgment. 

This  alternative  structure  requires  depicting  two  STFs  (and  hence 
two  paths)  for  the  immediate  targeting  experts  in  experimental  unit  1. 

An  alternative  structural  hypothesis  for  the  Air  Force  targeteers 
(experimental  unit  2  in  Fig.  1)  might  be  that  they  combine  information 
about  enemy  emitters  separately  from  information  about  enemy  vehicles; 
then  they  take  the  values  of  those  outputs  and  combine  them  with  their 
value  associated  with  the  processing  capability.  This  alternative 
structural  hypothesis  that  requires  3  STFs  for  targeteering  experts  is 
depicted  in  the  lower  portion  of  Fig.  2.  Thus,  this  entire  structural 
alternative  for  these  two  groups  of  experts  produces  a  representation 
with  five  STFs  and  hence  five  paths--two  for  experimental  unit  1  and  3 
for  experimental  unit  2.  Other  clumping  hypotheses  would  change  the 
number  of  paths  and  STFs.  For  example,  the  hypothesis  that  emitter 
information  was  not  perceptually  clumped  but  vehicle  information  was  (or 
vise  versa)  would  reduce  experimental  unit  2  to  two  2  STFs,  and  the 
entire  structure  to  4  STFs. 

The  second  reason  for  hypothesizing  alternative  structures  is  that 
too  many  motors  define  a  single  experimental  unit.  This  presents  a 
problem  b<-..ause  the  major  expei  i mental  design  feature  needed  to  test 
judgment  models  (STFs)  is  the  .artorial  combination  of  stimuli.  Each 
c.ell  in  the  factori.il  design  translates  into  a  questionnaire  item.  A 
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fully  crossod  design  generates  quest iona i re  items  that  contain  as  many 
pieces  of  information  as  factors  used  in  the  design.  1'or  example,  three 
factors  would  produce  a  questionnaire  item  with  three  piet.es  of 
information.  Our  research  has  indicated  that  between  five  and  seven 
pieces  of  information  (depending  on  the  interrelationships  among  the 
factors)  are  maximum  for  a  respondent  to  simultaneously  process.  Miller 
(23)  has  estimated  seven  plus  or  minus  two  pieces  of  information  to  be 
maximum . [ 1 ) 

The  number  of  factors  defining  an  experimental  unit  can  be  reduced 
by  hypothesizing  a  subset  of  the  factors  to  impact  on  a  suboutcome  that 
is  meaningful  within  the  framework  of  the  evaluation  goals.  For 
example,  the  execution  status  information  factor  shown  in  Fig.  2  could 
be  changed  from  an  intermediary  factor  to  a  suboutcome.  This  would 
require  defining  the  factor  along  a  dimension  that  is  meaningful  both 
for  the  respondent  group  that  that  would  judge  that  factor  as  an 
independent  variable  (the  immediate  targeting  experts  for  the  structure 
shown  in  Fig.  2)  and  the  respondent  group  that  would  judge  the  factor  as 
a  dependent  variable  (this  could  be  the  same  or  another  expert  respndent 
group  entirely).  In  situations  where  the  same  respondent  group  is 
expert  in  two  experimental  units,  they  would  be  asked  to  make  judgments 
about  their  two  different  tasks  at  different  times,  or  different  subsets 
of  the  respondents  would  be  assigned  to  the  different  experimental 

[1]  Questionnaire  length  also  increases  rapidly  as  the  number  of 
factors  (and  factor  levels)  are  added  to  the  experimental  unit  when  fac¬ 
torial  designs  are  used.  Variations  on  completely  crossed  factorial 
designs  that  descrease  the  questionnaire  length  while  maintaining  enough 
constraints  to  adequately  test  among  the  unique  predictions  of  the  func¬ 
tions  being  considered  are  discussed  and  illustrated  in  Birnbaum  and 
Stegner  (12)  and  Veit  et  al.  (39). 
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units. 

Figure  1  was  selected  to  illustrate  the  ideas  behind  developing  a 
complex  system  structure  and  hypothesizing  alternative  structures 
because  it  only  had  two  experimental  units.  In  reality,  most  complex 
systems  will  be  quite  a  bit  larger.  For  example,  a  tactical  Air  Force 
command  and  control  system  researched  by  Rand  was  composed  of  105 
factors  to  be  manipulated,  30  suboutcomes ,  and  one  final  system  outcome. 


STF  HYPOTHESES 


Table  2  describes  some  algebraic  functions  that  might  be 
entertained  as  STFs  at  the  outset  of  a  complex  system  investigation. 
Ideas  about  what  functions  to  entertain  come  from  the  judgment 
literature  and  previous  research  in  the  problem  domain  of  interest. 

The  functions  in  Table  2  have  been  specified  for  "f"  factors;  i 
ref ers  to  the  factor,  and  j  to  the  factor  level;  w(0)and  s(0)  are 
"initial  estimate"  parameters - -what  the  response  would  be  in  the  absence 
of  specific  information,  and  r  is  the  subjective  response. [2] 

[2 |  The  J  function  shown  m  Eq .  3  that  relates  subjective 
responses,  r,  to  observed  responses  is  not  indicated  in  these  equations. 
Its  determination  is  discussed  in  literature  on  the  scale-free  design 
(5,  14). 

The  functions  shown  in  Table  2  have  both  scale  value  (s)  and  weight 
(w)  parameters  For  all  nut  the  differential  weight  function  shown  in 
Pane!  B,  weights  are  associated  with  factors  and  are  constant  across 
factor  levels.  The  di i I eiviit ial  weight  function  allows  a  different 
weight  (and  scale  value)  for  each  factor  level.  The  functional  descrip¬ 
tion  "Multiplicative  combination  of  factors"  at  the  top  of  Panel  B  al¬ 
lows  for  a  wide  variety  of  algebraic  formulations  where  all  factors  may 
combine  mu  1 1 i p 1 i cat i ve 1 y ,  or  two  or  more  factors  may  combine  multiplica¬ 
tive);,  but  add i t i ve 1 y  with  lie*  other  (actors.  (An  example  with  three 
factors.  A,  B,  and  C,  where  factors  A  and  I)  combine  multiplicat ivcly  but 
they  both  combine,  addi  lively  with  C  would  be : 


r  =(w  AH  +  w  C)/(w  +  w  ). 

1  2  12 


(A) 
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Each  function  described  in  Table  2  makes  a  di f f erent  prediction 
with  respect  to  the  pattern  the  judgment  data  should  follow  when 
appropriate  experimental  designs  are  used.  For  example,  one  prediction 
all  of  the  functions  shown  in  the  left-hand  panel  have  in  common  is  that 
of  no  interactions  among  the  factors  Conversely,  the  functions  in  the 
right-hand  panel  can  account  for  interactions  among  the  factors.  The 
functions  within  each  panel  make  other  differential  predictions  with 
respect  to  the  judgment  data. 

It  is  important  that  both  structural  and  STF  hypotheses  be 
specified  in  advance  for  each  experimental  unit  so  that  experimental 
designs  allow  tests  among  their  predictions. 


I 
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IV.  DETERMINING  STFs  AND  FINAL  STRUCTURE 


The  explanatory  power  of  a  proposed  STF  lies  in  its  ability  to 
reproduce  the  systematic  details  of  the  data.  A  good  way  to  assess  this 
ability  is  to  plot  predicted  points  and  responses  on  the  same  ordinate 
as  a  function  of  the  levels  of  one  (or  more)  factor  with  a  separate 
curve  for  each  level  of  another  factor.  These  predicted  and  obtained 
graph  allows  assessment  of  the  magnitude  and  direction  of  the  function's 
deviations.  Statistical  tests  among  the  viable  subset  of 
structure/ function  combinations  are  made  for  each  experimental  unit 
using  STEPIT,  a  parameter-estimation  subroutine  (15)  programmed  to 
select  parameters  that  minimize  the  sum  of  squares  discrepancies  between 
the  data  and  the  STF's  predictions.  The  STF  with  the  smallest 
discrepancy  (taking  diferences  in  degrees  of  freedom  into  account)  would 
be  considered  the  "best-fit"  STF  for  that  experimental  unit.  However, 
if  predicted  and  obtained  graphs  reveal  that  deviations  are  large  and 
systematic  for  the  statistically  "best"  function,  that  function  would 
also  be  rejected  as  the  appropriate  STF.  Predicted  and  obtained  graphs 
provide  a  tool  for  diagnosing  the  pattern  of  the  residuals  (errors)  and 
determining  a  "correct"  function.  If  a  new  funt  'on(s)  suggested  by  the 
pattern  of  deviations  cannot  be  adequately  tested  on  the  available  data 
(i.e.,  variations  on  the  original  experimental  design  are  needed  to 
adequately  test  its  predictions),  it  would  be  necessary  to  redesign  the 
experiment  and  collect  new  data. 

The  fit  of  the  STFs  to  the  data  drives  the  structure.  For  example, 


the  structure  shown  in  the  upper  portion  of  Fig.  2  postulates  that  an 


STF  that  explains  the  effects  of  Alert  forces,  Airborne  forces,  and 
Weather  on  Execution  status  information  is  embo-dded  in  an  overall  STF 
that  explains  the  effects  of  the  four  Immediate  Targeting  Factors-- 
Target  Identification,  Facility  Operability,  Execution  Status 
Information,  and  Disseminat ion .  Figure  1  postulates  that  immediate 
targeting  judgments  are  better  explained  by  a  simpler  six  factor  STF. 

If  a  simple  six  factor  STF  provided  a  good  fit  to  the  data,  the 
immediate  targeting  portion  of  the  structure  would  be  depicted  as  shown 
in  Fig.  1.  Thus,  the  STF  drives  the  decision  on  how  the  structure 
should  be  depicted  as  well  as  the  subjective  values  needed  to  evaluate 
and  compare  systems. 


i 
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V.  COMPARING  SYSTEMS 

Once  the  system's  structure  and  STFs  have  been  determined,  it  is 
possible  to  compare  systems  that  are  defined  by  the  same  structure  but 
differ  in  capability  levels.  Capabilities  that  are  input  to  a  system 
are  defined  in  terms  of  the  system's  primitive  factor  levels;  sytems 
that  are  different  differ  in  at  least  one  primitive  factor  level. 
Determining  the  output  of  a  single  system  requires  putting  the 
subjective  scale  values  associated  with  each  primitive  factor  level 
defining  the  system  into  the  STFs  associated  with  the  primitive  factor, 
computing  those  STFs,  transfering  the  outputs  from  those  STFs  into  the 
STFs  to  which  they  are  linked  (after  converting  to  subjective  scale 
values),  and  so  forth  until  all  STFs  have  been  computed  and  the  final 
outcome  measure  has  been  obtained. 

An  example  of  this  evaluation  procedure  is  presented  in  this 
section.  The  structure  shown  in  Fig.  3  is  the  final  structure  that 
resulted  from  the  structural  hypotheses  entertained  by  Veit  et  al., 
(38).  (Note  that  the  difference  in  structure  between  Figs.  2  and  3  is 
that  the  intermediary  factor  "Execution  Status"  has  been  omitted.)  The 
function  selected  as  the  appropriate  STF  is  named  at  each  path.  At  the 
top,  a  range  model  (see  Table  2)  best  accounted  for  the  immediate 
targeting  experts'  data.  The  minus  sign  indicates  that  the  omega  term 
in  the  range  model  was  negative,  indicating  overall  divergent 
interactions  among  the  6  factors  in  this  experimental  unit.  The 
interpretation  of  the  divergent  interactions  is  that  the  better  the 
capability  that  the  immediate  targeting  expert  has  to  work  with  on  one 
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factor,  the  more  of  a  difference  it  makes  how  good  their  capabilities 
are  on  the  other  factors.  A  range  model  with  a  positive  omega  term  best 
accounted  for  the  targeteers  data  at  the  target  identification  path. 

The  convergent  interaction  found  here  indicated  that  the  more  the 
targeteers  knew  about  enemy  emitters,  the  less  of  a  difference  it  made 
about  how  much  information  they  had  on  enemy  vehicles  (and  vise  versa) . 
(Better  capabilities,  however,  always  received  a  higher  judgment).  The 
two  intermediary  factors,  Vehicles  and  Emitters,  hypothesized  in  Fig.  2 
were  retained.  At  the  vehicle  path,  the  range  model  with  a  negative 
omega  term  best  accounted  for  overall  divergent  interactions  found  among 
these  three  factors.  The  relative  weight  function  with  an  initial 
impression  (the  third  function  shown  in  Table  2A)  best  accounted  for  the 
emitter  data;  that  is,  the  overriding  trend  in  these  data  was 
independence  among  the  factors  on  the  target  identification  judgments. 

Suppose  it  was  of  interest  to  compare  three  different  systems 
described  by  the  structure  shown  in  Fig.  3  on  how  well  the  immediate 
targeting  people  thought  they  could  do  their  job  (the  percent  force 
application  opportunities  they  thought  they  could  exploit).  As  stated 
earlier,  a  particular  system  is  identified  by  its  primitive  factor 
levels .  The  primitive  factors  in  the  structure  shown  in  Fig.  3  are  the 
location/classification,  coverage,  and  currency  factors  associated  with 
vehicles;  the  location,  coverage  and  currencey  factors  associated  with 
enemy  emitters;  Processing;  Facility  operability;  Alert  forces;  Airborne 
forces;  Weather;  and  Dissemination.  (Definitions  of  these  factors  were 
presented  in  Table  1.)  Three  different  systems  are  shown  in  Figs.  4-6. 
The  circled  primitive  factor  levels  identify  their  differences.  The 
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first  thing  that  is  needed  for  an  evaluation  of  these  systems  is  the 
subjective  values  associated  with  these  levels. 

We  will  describe  how  subjective  values  are  obtained  for  primitive 
factor  levels  and  how  STFs  are  compute!  using  the  factors  and  range 
model  at  the  vehicle  path  shown  in  Fig.  U.  For  this  path,  subjective 
values  associated  with  a  currency  level  of  30  minutes,  a  coverage  level 
of  40°<>  and  a  location/classification  level  of  "Clr.wx.  loc"  (can  locate 
and  classify  enemy  vehicles  in  clear  weather  only)  are  needed  to  compute 
the  range  model.  In  Figs.  7A-7C,  subjective  scale  values  are  plotted  as 
a  function  of  the  factor  levels  for  Vehicle  Location/Classification, 
Vehicle  Coverage,  and  Vehicle  Currency,  respect ive ly ; ( 1 ]  the  range  model 
for  the  vehicle  path  is  written  at  the  top  of  Fig.  7.  For  Figs.  7B  and 
7C,  the  resulting  curves  represent  the  psychophysical  functions  (utility 
curves)  for  these  factors  (H  in  Eq .  1).  The  subjective  scale  value 
associated  with  a  currency  level  of  30  minutes  is  show'n  by  the 
connecting  dashed  line  in  7C.  When  factors  are  defined  along  a  physical 
continuum  (e.g.,  time,  percent,  distance),  it  is  possible  to  estimate 
scale  values  of  factor  levels  not  previously  identified  or  manipulated 
by  projecting  from  the  psychophysical  function.  This  is  demonstrated 
for  the  40°o  coverage  value  in  Fig.  7B.  When  factor  levels  are  written 
descriptions  such  as  the  location/classification  factor  (Fig.  7A) ,  scale 
values  are  obtained  only  for  the  descriptions  used  in  the  experiment. 

The  scale  value  needed  for  the  system  described  in  Fig.  4  is  indicated 
on  the  graph.  If  a  di f ferent  written  description  (one  not  used  in  the 

(1|  Recall  that,  when  the  STF  has  been  determined,  the  parameter 
values  are  known;  they  are  the  least-squares  estimates  of  the  function. 


22 


experiment)  is  used  to  define  a  primitive  factor  level,  it  lias  to  be 
scaled  in  a  new  experiment  for  its  experimental  unit,  since  a  curve 
cannot  be  drawn  to  connect  scale  values  associated  witli  written 
descriptions.  The  subjective  values  shown  in  Fig.  7  and  the  other 
parameters  of  the  range  function  would  be  aggregated  as  specified  by  the 
function  to  obtain  the  output  of  the  intermediary  vehicle  factor.  This 
output  is  used  in  the  range  model  with  a  positive  omega  coefficient  at 
the  immediate  targeting  path.  This  process  is  repeated  for  each  STF , 
starting  at  the  primitive  component  levels  and  proceeding  to  the  top 
(immediate  targeting  output). 

For  the  particular  system  shown  in  Fig.  4,  the  theoretical 
prediction  is  that  33°0  of  the  important  targets  would  be  identified, 
which  leads  to  a  theoretical  prediction  that  about  48°„  of  the  important 
immediate  targeting  opportunities  would  be  exploited.  Figure  5  shows 
that  increasing  the  targeteers'  ability  to  identify  targets  to  68°0  (by 
changing  the  systems'  capabilities  as  indicated  by  the  circled  primitive 
factor  levels)  increases  the  ability  to  do  immediate  targeting  to  only 
52°o,  keeping  the  other  capabilities  for  this  upper  portion  of  the 
structure  the  same  as  in  Fig.  4.(2)  In  Fig.  6,  the  primitive  factor 
levels  for  the  lower  portion  of  the  structure  are  the  same  as  in  Fig.  4, 
but  the  immediate  targeting  capabilities  have  been  greatly  improved  in 
several  areas  (Alert  Forces,  Airborne  Forces,  and  1)  i ssern  inn t i on ) .  [ 3 ) 

[2]  An  interpretation  nt  these  results  is  1 1 1  it  the  people  training 
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With  those  changes,  the  theoretical  prediction  is  that  59°,.  of  the 
immediate  targeting  oppor tun i t i es  would  he  exploited. 

Tradeoffs  in  the  contribution  of  two  factors  to  an  outcome  can  he 
assessed  by  looking  at  i  graph  oi  theoretical  predictions.  This  is 
shown  in  Fig.  8  for  the  facility  operability  and  dissemination  factors 
of  Figs.  5-6.  It  car,  be  seen  from  this  graph  that  the  prediction  'or. 
the  y-axis)  is  about  the  same  for  a  dissemination  level  of  1  Oh,  and  a 
Facility  operability  level  of  90°„  as  for  a  dissemination  level  of  60% 
and  a  Facility  operability  level  of  30%.  Other  tradeoffs  between  these 
factors  can  be.  assessed  by  drawing  horizontal  lines  through  the 
theoretical  curves.  For  graphic  tradeoffs  among  3  factors,  a  graph  for 
two  factors  such  as  that  shown  in  Fig.  8  would  be  plotted  at  each  level 
of  the  third  factor.  Graphic  procedures  for  evaluating  tradeoffs  would 
be  especially  useful  in  situations  where  the  decision  about  which  system 
changes  to  make  involve  only  a  few  system  factors. 

The  actual  primitive  factor  levels  selected  in  an  evaluation  would 
be  determined  from  such  things  as  systems  being  entertained  for 
purchase,  production,  development,  and/or  present  capability  levels. 
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VI  .  SUMMARY  REMARKS 

Illustrations  in  this  paper  have  been  in  the  Air  I'orce  command  and 
control  arena,  since  that  is  the  problem  domain  to  which  the  approach 
has  been  applied.  However,  it  is  hoped  that  the  demonstrations  served 
to  illustrate  how  the  approach  could  be  applied  to  a  wide  variety  of 
systems.  System  evaluations  obtained  using  this  approach  could  be  used 
to  aid  decisionmakers  in  assessing  operational  consequences  of 
alternative  system  capabilites,  since  they  serve  to  pinpoint  where  in 
the  system  changes  make  a  difference.  Further  it  aids  in  decisions 
about  what  research  to  support  as  well  as  equipment  to  develop,  produce, 
and/or  purchase. 

The  major  advantage  of  using  the  STF  approach  to  analyze  systems  is 
that  the  tested  STF  provides  a  validity  base  for  conclusions  about  what 
affects  system  outcomes.  This  testability  feature  is  absent  from  a 
number  of  commonly  used  approaches  to  analyzing  systems  described 
earlier.  However,  it  is  vital  feature  if  conclusions  are  to  be 
credible . 


«.#v 
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